Sparse Covariance Coding

Recently, there has been considerable interest in employing unsupervised learning methods as feature extractors for supervised learning tasks such as classification. The literature shows that methods based on this approach have proved to be competitive with established state-of-the-art machine learning strategies. One important recent advance was the discovery by (Hinton, Osindero, & Teh, 2006) of the role that unsupervised learning methods can play as an initialization for subsequent training for a supervised task. Unsupervised learning methods that extract sparse features of the data have received particular attention and has been shown by Raina, Battle, Lee, Packer, and Ng (2007) to significantly improve classification performance. While there are some clear advantages of not including label information in learning the feature set or initialization — for instance, Raina et al. (2007) use a sparse coding scheme trained on a large quantity of “related” unlabeled data to augment the feature set training — the use of unsupervised feature extraction gives no safeguard against highly discriminative features being cast aside. In relatively uncomplicated tasks, such as classifying hand-written digits using the MNIST dataset (Lecun, Bottou, Bengio, & Haffner, 1998), the lack of supervisory information in determining the representation is not likely to present a problem as most of the salient features of the image are useful in the classification task. However in more complex tasks where a large number of salient features of the data can have nothing to do with the target task, unsupervised learning methods are not likely to efficiently generate discriminative features. On the other hand, the lesson of the utility of extracting features of the unsupervised input data pattern, demonstrated in Hinton et al. (2006) and in Raina et al. (2007) should not be ignored. In this work, we focus on the problem of learning a sparse representation of data that stakes out a compromise position: explicitly taking into account information such as the labels in a classification task, while simultaneously attempting to capture descriptive features of the input data. Our method is based on a novel probabilistic interpretation of the canonical ridge analysis (Vinod, 1976), a regularized version of canonical correlation analysis.